89 research outputs found
Robust Latent Representations via Cross-Modal Translation and Alignment
Multi-modal learning relates information across observation modalities of the
same physical phenomenon to leverage complementary information. Most
multi-modal machine learning methods require that all the modalities used for
training are also available for testing. This is a limitation when the signals
from some modalities are unavailable or are severely degraded by noise. To
address this limitation, we aim to improve the testing performance of uni-modal
systems using multiple modalities during training only. The proposed
multi-modal training framework uses cross-modal translation and
correlation-based latent space alignment to improve the representations of the
weaker modalities. The translation from the weaker to the stronger modality
generates a multi-modal intermediate encoding that is representative of both
modalities. This encoding is then correlated with the stronger modality
representations in a shared latent space. We validate the proposed approach on
the AVEC 2016 dataset for continuous emotion recognition and show the
effectiveness of the approach that achieves state-of-the-art (uni-modal)
performance for weaker modalities
Multiple Source Localization Based on Acoustic Map De-Emphasis
This paper describes a novel approach for localization of multiple sources overlapping in time. The proposed algorithm relies on acoustic maps computed in multi-microphone settings, which are descriptions of the distribution of the acoustic activity in a monitored area. Through a proper processing of the acoustic maps, the positions of two or more simultaneously active acoustic sources can be estimated in a robust way. Experimental results obtained on real data collected for this specific task show the capabilities of the given method both with distributed microphone networks and with compact arrays
Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms
Outdoor acoustic events detection is an exciting research field but
challenged by the need for complex algorithms and deep learning techniques,
typically requiring many computational, memory, and energy resources. This
challenge discourages IoT implementation, where an efficient use of resources
is required. However, current embedded technologies and microcontrollers have
increased their capabilities without penalizing energy efficiency. This paper
addresses the application of sound event detection at the edge, by optimizing
deep learning techniques on resource-constrained embedded platforms for the
IoT. The contribution is two-fold: firstly, a two-stage student-teacher
approach is presented to make state-of-the-art neural networks for sound event
detection fit on current microcontrollers; secondly, we test our approach on an
ARM Cortex M4, particularly focusing on issues related to 8-bits quantization.
Our embedded implementation can achieve 68% accuracy in recognition on
Urbansound8k, not far from state-of-the-art performance, with an inference time
of 125 ms for each second of the audio stream, and power consumption of 5.5 mW
in just 34.3 kB of RAM
Learning to Rank Microphones for Distant Speech Recognition
Fully exploiting ad-hoc microphone networks for distant speech recognition is
still an open issue. Empirical evidence shows that being able to select the
best microphone leads to significant improvements in recognition without any
additional effort on front-end processing. Current channel selection techniques
either rely on signal, decoder or posterior-based features. Signal-based
features are inexpensive to compute but do not always correlate with
recognition performance. Instead decoder and posterior-based features exhibit
better correlation but require substantial computational resources. In this
work, we tackle the channel selection problem by proposing MicRank, a learning
to rank framework where a neural network is trained to rank the available
channels using directly the recognition performance on the training set. The
proposed approach is agnostic with respect to the array geometry and type of
recognition back-end. We investigate different learning to rank strategies
using a synthetic dataset developed on purpose and the CHiME-6 data. Results
show that the proposed approach is able to considerably improve over previous
selection techniques, reaching comparable and in some instances better
performance than oracle signal-based measures
Unsupervised cross-modal deep-model adaptation for audio-visual re-identification with wearable cameras
Model adaptation is important for the analysis of audio-visual data from body worn cameras in order to cope with rapidly changing scene conditions, varying object appearance and limited training data. In this paper, we propose a new approach for the on-line and unsupervised adaptation of deep-learning models for audio-visual target re-identification. Specifically, we adapt each mono-modal model using the unsupervised labelling provided by the other modality. To limit the detrimental effects of erroneous labels, we use a regularisation term based on the Kullback-Leibler divergence between the initial model and the one being adapted. The proposed adaptation strategy complements common audio-visual late fusion approaches and is beneficial also when one modality is no longer reliable. We show the contribution of the proposed strategy in improving the overall re-identification performance on a challenging public dataset captured with body worn cameras
An Experimental Review of Speaker Diarization methods with application to Two-Speaker Conversational Telephone Speech recordings
We performed an experimental review of current diarization systems for the
conversational telephone speech (CTS) domain. In detail, we considered a total
of eight different algorithms belonging to clustering-based, end-to-end neural
diarization (EEND), and speech separation guided diarization (SSGD) paradigms.
We studied the inference-time computational requirements and diarization
accuracy on four CTS datasets with different characteristics and languages. We
found that, among all methods considered, EEND-vector clustering (EEND-VC)
offers the best trade-off in terms of computing requirements and performance.
More in general, EEND models have been found to be lighter and faster in
inference compared to clustering-based methods. However, they also require a
large amount of diarization-oriented annotated data. In particular EEND-VC
performance in our experiments degraded when the dataset size was reduced,
whereas self-attentive EEND (SA-EEND) was less affected. We also found that
SA-EEND gives less consistent results among all the datasets compared to
EEND-VC, with its performance degrading on long conversations with high speech
sparsity. Clustering-based diarization systems, and in particular VBx, instead
have more consistent performance compared to SA-EEND but are outperformed by
EEND-VC. The gap with respect to this latter is reduced when overlap-aware
clustering methods are considered. SSGD is the most computationally demanding
method, but it could be convenient if speech recognition has to be performed.
Its performance is close to SA-EEND but degrades significantly when the
training and inference data characteristics are less matched.Comment: 52 pages, 10 figure
Leveraging Speech Separation for Conversational Telephone Speaker Diarization
Speech separation and speaker diarization have strong similarities. In
particular with respect to end-to-end neural diarization (EEND) methods.
Separation aims at extracting each speaker from overlapped speech, while
diarization identifies time boundaries of speech segments produced by the same
speaker. In this paper, we carry out an analysis of the use of speech
separation guided diarization (SSGD) where diarization is performed simply by
separating the speakers signals and applying voice activity detection. In
particular we compare two speech separation (SSep) models, both in offline and
online settings. In the online setting we consider both the use of continuous
source separation (CSS) and causal SSep models architectures. As an additional
contribution, we show a simple post-processing algorithm which reduces
significantly the false alarm errors of a SSGD pipeline. We perform our
experiments on Fisher Corpus Part 1 and CALLHOME datasets evaluating both
separation and diarization metrics. Notably, without fine-tuning, our SSGD
DPRNN-based online model achieves 12.7% DER on CALLHOME, comparable with
state-of-the-art EEND models despite having considerably lower latency, i.e.,
50 ms vs 1 s.Comment: Submitted to INTERSPEECH 202
- …